组合测试方法：配对测试实践

实施组合测试

常用的Pairwise工具集：http://www.pairwise.org/tools.asp

成对测试（Pairwise Testing）又称结对测试、两两测试，是一种正交分析的测试技术。

成对组合覆盖这一概念是Mandl于1985年在测试Aad编译程序时提出来的。是当不可能遍历产品所有功能点时用到的一种测试手段，即如果用尽可能少的工作发现最多的缺陷（测试投入产出性价比）。

经验证，Cohen等人应用成对组合覆盖测试技术对Unix中的“Sort”命令进行了测试。测试结果表明覆盖率高达90%以上，可见成对成对测试是一种非常有效的测试用例设计方法。

成对组合覆盖要求任意两个因素（输入条件）的所有水平组合至少要被覆盖1次。实现成对组合覆盖的算法工具有很多，其中微软的PICT就是其中一个。

测试用例生成工具 —— PICT(Pairwise Independent Combinatorial Testing tool)

PICT 可以有效地按照两两测试的原理,进行测试用例设计.在使用PICT时,需要输入与测试用例相关的所有参数,以达到全面覆盖的效果。

常用的Pairwise工具集：http://www.pairwise.org/tools.asp 处下载PICT工具。

开源仓库：https://github.com/microsoft/pict

Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios

Pairwise Testing

Introduction to Pairwise Testing

t-wise testing is t-wise testing with a thoroughness of t.

The term all-pairs testing is simply equivalent to pairwise testing.

Mixed-strength testing is to mix, for example, 2-wise and 3-wise testing in the same round of testing.

Combinatorial testing is a more general term that includes pairwise testing but also other testing techniques.

Combinatorial testing > Pairwise testing > Orthogonal array testing

Pairwise testing fits nicely into the testing phase of software or hardware development. It works well with agile development practices.

Pairwise testing is usually not fully automatic. Pairwise testing usually requires some manual work to describe the thing that is to be tested. 

Usually a pairwise testing tool then takes this description (or model) and generates the pairwise tests automatically.

Pairwise testing is more focused on putting your system into varied situations.

Having a domain expert make tests for a software or hardware system is valuable and highly recommended.

Pairwise testing is not a replacement for this kind of testing but complementary to it.

Pairwise tests acts in a similar way, it explores the different paths of interaction.

From a user’s point of view, pairwise testing is light-weight and simple to apply to even the larger cases.

From the pairwise tool developer’s point of view, however, pairwise testing involves some heavy-weight computations. This was until the last few years a bottleneck in the application of pairwise testing to the larger real-life cases, but in recent years effective algorithms for pairwise testing has been invented and are now available in commercial products.>>> Some papers on the computations.

The effectiveness of pairwise testing is neither hypothetical nor only academic. The effectiveness of pairwise testing has been studied by looking at existing large-scale software or hardware and their histories of problems and bugs. As the thoroughness of pairwise testing increases, it has been found that increasing amounts of bugs can be found. I one large study, it was found that for a thoroughness of 1, 2 and 3, the percentage of bugs that might be found was 50, 75 and 90%, respectively. This study found no bugs that required more than a thoroughness of 6, a result that has stood the test of time. This is the reason why a thoroughness of 6 is usually the maximum considered.

As the thoroughness of pairwise testing increases, so does the effort required to do the tests. A thoroughness of 2, i.e. 2-wise or pairwise, has a very good tradeoff between thoroughness and effort. A thoroughness of 3 is a good for more important rounds of testing, for example, for later stages of development.

For testing highly critical software of hardware, pairwise testing is a good start, but other quality assurance techniques should be used in addition.

A pairwise test suite might contain, for example, 30 test cases. The accumulated coverage after each test case has been added will typically form a curve like this:

As we can see in this example, the first 12 tests include more than 95% of the pairs. So, the last 18 tests cover less than 5%.

In ordinary pairwise test generation, the pairs included (pseudo-)randomly, meaning that any faulty pair is most likely to be included early in the test suite. Thus, a partial test suite of, for example, 95% can be expected to detect 95% of the bugs that a complete pairwise test suite would find.

Pairwiser(Web-based, free & commercial)

https://inductive.no/pairwiser/knowledge-base/testing-example-1/

Pairwise testing is not well suited for a testing problem with a few parameters with relatively many combinations. As we will see in other examples, pairwise testing performs very well when we have more parameters, which the most interesting, real industrial cases surely tend to have!

Pairwise testing was combined with a kind of equivelence class selection technique to reduce the number of values to include for generating the pairwise tests. This is a quite common technique to combine with pairwise testing.

https://inductive.no/pairwiser/knowledge-base/testing-example-2/

Total inputs:

CombTestWeb(Web-based, free)

>>> Paper: A framework and a web implementation for combinatorial testing